Skip to content

Feature/storage/monitoring tool#1248

Merged
ihsaan-ullah merged 13 commits into
developfrom
monitoring-tool
Dec 14, 2023
Merged

Feature/storage/monitoring tool#1248
ihsaan-ullah merged 13 commits into
developfrom
monitoring-tool

Conversation

@Didayolo

@Didayolo Didayolo commented Nov 28, 2023

Copy link
Copy Markdown
Member

Merge the PR into a branch in our own codalab/codabench repository.

Original PR:


Description

This PR aims to offer a monitoring tool similar to the one available in Codalab.
The following features have been developed:

  • The monitoring tool is only accessible by admins
  • It is accessible under the user menu -> analytics -> storage tab
  • 3 metrics are available
    • Usage history
    • Competitions usage history and details
    • Users usage history and details
  • Metrics can be download using the download buttons
  • The metrics only displays what has been computed by the server beforehand
  • 2 background tasks has been coded:
    • a reset task that get executed every 3 month (based on the value that is set in the settings/base.py). It reset the value of all computed file sizes so they will be re-computed again without any shifting on the next run of the storage analytics task
    • a storage analytic snapshot task that get executed every week (based on the value that is set in the settings/base.py). It measures all files with unset size. Then it measures the storage usage per category (competition, user or admin) and per day. It then check for database <-> storage inconsistency (and created a log file in the logs/ folder). And save everything into the database so it becomes query-able by the front-end.

Issues this PR resolves

#713

Checklist

  • Code review by me
  • Hand tested by me
  • I'm proud of my work
  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

@Didayolo

Copy link
Copy Markdown
Member Author

@ihsaan-ullah ihsaan-ullah merged commit 0311135 into develop Dec 14, 2023
@ihsaan-ullah ihsaan-ullah deleted the monitoring-tool branch December 14, 2023 15:11
@ihsaan-ullah ihsaan-ullah mentioned this pull request Jan 25, 2024
43 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants